Do ‘ Faculty of 1000 ’ ( F 1000 ) ratings of ecological publications serve as reasonable predictors of their future impact ?

There is an increasing demand for an effective means of post-publication evaluation of ecological work that avoids pitfalls associated with using the impact factor of the journal in which the work was published. One approach that has been gaining momentum is the ‘Faculty of 1000’ (hereafter F1000) evaluation procedure, in which panel members identify what they believe to be the most ‘important’ recent publications they have read. Here I focused on 1530 publications from 7 major ecological journals that appeared in 2005, and compared the F1000 rating of each publication with the frequency with which it was subsequently cited. The mean and median citation frequencies of the 103 publications highlighted by F1000 was higher than for all 1530 publications, but not substantially so. Further, the F1000 procedure did not highlight any of the 11 publications that were each cited over 130 (and up to 497) times, while it did highlight 14 publications that were each cited between 4 and 9 times. Further, 46% and 31% of all manuscripts highlighted by F1000 were cited less often than the mean and median respectively of all 1530 publications. Possible reasons for the F1000 process failing to identify high impact publications may include uneven coverage by F1000 of different ecological topics, cronyism, and geographical bias favoring North American publications. As long as the F1000 process cannot identify those publications that subsequently have the greatest impact, it cannot be reliably used as a means of post-publication evaluation of the ecological literature. The past decade has seen a substantial increase in the use of external evaluations to evaluate scientific performance, ranging from the level of the individual to university departments and even whole institutions, and frequently these form the basis for major decisions about allocation of funding and other resources. As the scientific publication is the most tangible output of the scientific process, many evaluations have focused on assessing the relative ‘quality’ or ‘merit’ of publications in terms of their likely impact in the field. The true impact of a publication does not usually become apparent until several years after it is published, especially in the case of ecological research. However, evaluations often focus on the most recent publications produced by an individual or organization, because evaluations usually focus on current outputs and thus future potential, rather than on publications from several years previously. As such, there is an obvious need for assessments of ‘quality’ that can be measured soon after papers are published without waiting for them to garner citations or recognition. Any short term measure of scientific merit has potential drawbacks and may be potentially influenced by various types of bias (Lortie et al. 2007). One such short-term measure used in many evaluations involves rating publications by the impact factor of the journals in which they appear. However, there are numerous criticism of this focus on impact factors (Seglen 1997), the most important arguably being that most publications in high impact factor journals still only attract modest numbers of citations (Seglen 1997, Leimu and Koricheva 2005). The increasing importance being placed on journal impact factors has led some editors to concentrate on enhancing their journal’s impact factor (sometimes in rather